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Abstract  Assessing  the  health  and  environmental  risks  of 
engineered  nanomaterials  (ENMs)  continues  to  be  a  chal¬ 
lenging  endeavor.  Due  to  extensive  challenges  related  to 
applying  traditional  risk  assessment  frameworks  to  ENMs, 
decision  making  regarding  the  use  of  ENMs  in  products 
and  applications  may  need  to  rely  on  structured  decision 
support  tools  such  as  risk  ranking  approaches.  This  study 
examines  the  use  of  one  risk  ranking  tool  that  incorporates 
both  quantitative  and  qualitative  information  regarding  the 
potential  human  health  risks  of  ENMs,  focused  primarily 
on  worker  and  soldier  health.  Using  a  case  study  involving 
Army  materiel  (i.e.,  equipment),  a  relative  risk  ranking 
algorithm  is  proposed  that  accounts  for  not  only  the 
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physicochemical  characteristics  of  the  ENMs,  but  also  the 
characteristics  of  the  Army  materiel.  In  this  way,  the 
resulting  risk  potential  for  soldiers  and  workers  is  not 
solely  based  on  the  inherent  characteristics  of  the  ENMs 
but  is  also  influenced  within  the  context  of  the  technology 
being  developed.  Among  other  important  findings,  the 
results  from  applying  this  risk  ranking  algorithm  in  this 
case  study  suggest  that  inhalation  from  accidental  expo¬ 
sures  to  carbon  nanotubes  and  copper  flakes  incorporated 
into  energy  and  obscurant  materiel  by  Army  workers  rank 
highest  relative  to  the  other  items  evaluated  in  this  baseline 
assessment.  As  the  presence  of  data  gaps  was  one  of  the 
greatest  challenges  to  applying  this  risk  ranking  algorithm, 
future  applications  may  benefit  from  reliance  on  a  contin¬ 
ually  revised  database  that  may  be  updated  in  real  time  and 
possibly  synced  with  publically  available  databases  in 
order  to  use  the  most  current  and  comprehensive  set(s)  of 
data  available. 

Keywords  Army  materiel  •  Engineered  nanomaterials  • 
Nanotechnology  •  Relative  risk  ranking  •  Risk  assessment 

1  Introduction 

Assessing  the  potential  human  health  and  environmental 
risks  of  engineered  nanomaterials  (ENMs)  within  the 
context  of  the  applications  and  products  in  which  they  are 
incorporated  continues  to  be  an  extremely  challenging 
endeavor.  A  decade  after  the  UK  Royal  Society  released 
their  2004  report  (Dowling  et  al.  2004),  scientists, 
researchers,  governments,  institutions,  and  industry  still 
grapple  with  the  multitude  of  uncertainties  and  data  gaps 
associated  with  characterizing  the  potential  ENM  hazards 
and  formulating  effective  strategies  to  handle  these 
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potential  risks.  Given  the  challenges  of  developing  suffi¬ 
cient  data  that  would  be  required  for  traditional  risk 
assessment  frameworks  (e.g.,  Choi  et  al.  2009),  risk 
assessors  are  continuing  to  refine  their  methods  and  tech¬ 
niques  to  perform  risk  assessments  using  combined  quan¬ 
titative  and  qualitative  frameworks  for  ENMs,  resulting  in 
various  alternatives  for  risk  analysis  (e.g.,  DuPont  2007; 
Tervonen  et  al.  2009;  Canis  et  al.  2010;  TUV/SUD  2012; 
Grieger  et  al.  2012).  Among  some  of  these  risk  analysis 
methods,  a  number  of  risk  ranking  tools  and  methods  have 
been  developed  and  demonstrated  for  ENMs  or  related 
processes  and  applications  (Tervonen  et  al.  2009;  Canis 
et  al.  2010;  Linkov  and  Seager  2011;  Linkov  et  al.  2013). 
The  use  of  risk  ranking  tools  may  be  particularly  advan¬ 
tageous  to  prioritize  materials  or  products  according  to 
their  risk  potential,  i.e.,  identify  the  “riskiest”  ENMs  or 
nano-products,  for  example,  for  further  research  or  inves¬ 
tigation.  This  may  be  useful  especially  in  cases  of  resource 
and  time  constraints. 

Based  on  the  research  conducted  by  the  US  Army 
Center  for  Environmental  Health  Research  (USACEHR) 
associated  with  the  incorporation  of  various  ENMs  into 
military  applications  (termed  “Army  materiel”),  this  ana¬ 
lysis  develops  and  implements  a  risk  ranking  tool  to  rank 
ENM-Army  materiel  pairs  based  on  their  relative  risks  to 
soldier  and  civilian  health  under  occupational  scenarios. 
The  USACEHR  has  been  conducting  research  and  devel¬ 
opment  efforts  on  the  incorporation  of  various  ENMs  into 
Army  materiel,  ranging  from  food  storage  to  computer 
systems  and  weaponry  (Kharat  et  al.  2006;  Turaga  et  al. 
2012).  With  such  a  wide  range  of  ENMs  being  incorpo¬ 
rated  into  a  variety  of  materiel,  the  US  Army  has  proac¬ 
tively  engaged  in  efforts  to  identify  the  ENMs  and  ENM- 
Army  materiel  applications  that  constitute  the  largest 
potential  health  risk  for  soldiers  using  the  materiel  and  for 
civilian  workers  developing  the  materiel. 

This  project  therefore  aimed  to  (1)  identify  and  inven¬ 
tory  the  ENMs  and  their  associated  applications  in  Army 
materiel,  (2)  develop  a  risk  ranking  algorithm  that  takes 
into  account  both  ENM  and  materiel  characteristics,  and 
(3)  apply  the  algorithm  and  associated  risk  ranking  tool  to 
prioritize  additional  assessments  based  on  the  human 
health  risk  potential  related  to  ENMs,  Army  materiel,  and 
ENM-materiel  pairs.  This  study  is  unique  in  that  real- 
world  ENM-Army  material  pairs  used  in  research  or  full- 
scale  field  applications  were  considered  to  perform  the 
relative  risk  ranking  rather  than  primarily  hypothetical  or 
pristine  ENMs.  Based  on  the  results  of  this  analysis,  rec¬ 
ommendations  are  provided  for  the  further  development  of 
risk  ranking  tools  specifically  for  ENMs  in  order  to  facil¬ 
itate  sound  decision  making  regarding  these  novel  materi¬ 
als  in  diverse  applications. 


2  Methodology 

2.1  Develop  inventory  of  ENMs  and  Army  materiel 

First,  an  inventory  of  ENMs  used  in  Army  research  and 
field  applications  (i.e.,  Army  materiel)  was  developed.  This 
inventory  was  incorporated  into  a  database  linking  the 
ENMs  and  materiel  applications  to  form  ENM-materiel 
pairs.  To  generate  an  inventory  of  ENMs  and  Army 
materiel,  16  subject  matter  experts  provided  by  the  US 
Army  were  interviewed  to  obtain  information  on  the 
identity,  use,  and  materiel  application  of  ENMs.  Telephone 
interviews  and  online  e-mail  questionnaires  were  used  to 
collect  information  on  the  ENMs  and  Army  materiel,  with 
the  completion  of  several  follow-up  interviews  and  online 
correspondence  to  confirm,  edit,  and  supplement  the 
inventory  with  additional  information  (See  Section  1  in 
Supplementary  Information  (SI)  for  questionnaires  used  in 
the  interview  process). 

2.2  Identification  of  ENM  characteristics 

After  producing  a  complete  inventory  of  ENMs  and  Army 
materiel  (available  in  Table  SI  in  SI),  the  physicochemical 
characteristics  of  the  ENMs  were  developed  for  risk  ranking 
purposes  (Table  1).  A  total  of  27  characteristics  of  ENMs 
were  used,  based  on  the  current  literature  pertaining  to  the 
health  and  environmental  risks  of  ENMs,  scientific  expertise 
in  ENM  behavior  and  toxicology,  information  provided  by 
the  Army,  and  general  consideration  of  potential  risks  for 
civilian  workers  and  soldiers  from  exposures  to  ENMs  (e.g., 
Thomas  and  Sayre  2005;  Boxall  et  al.  2007;  Auffan  et  al. 
2009;  Chappell  2009;  Oberdorster  2010).  For  organization 
purposes  as  well  as  for  later  use  for  risk  ranking,  these  ENM 
characteristics  were  grouped  into  five  different  categories 
which  describe  the  ENM  and/or  its  behavior:  fate,  reactivity, 
structure,  chemistry,  and  application-specific  characteristics. 
These  ENM  characteristics  and  groupings  all  relate  to  three 
risk  factors:  (1)  ENM  release  potential  (i.e.,  the  likelihood 
that  a  certain  ENM  concentration  or  mass  could  enter  an 
environmental  compartment,  such  as  air,  water,  soil/sedi¬ 
ment,  during  operational  or  accidental  use  of  the  Army 
materiel  in  which  it  is  embedded),  (2)  ENM  exposure 
potential  (i.e.,  the  likelihood  that  a  soldier  or  civilian  worker 
could  come  into  contact  via  dermal,  inhalation,  and/or 
ingestion  pathways  with  the  ENM  released  from  an  Army 
materiel,  and  (3)  ENM  toxicity  potential  (i.e.,  the  likelihood 
that  a  receptor  could  develop  adverse  effects  from  the 
release  and  subsequent  exposure  to  the  ENM  in  question). 
These  aforementioned  risk  factors  (i.e.,  ENM  release, 
exposure,  and  toxicity  potentials)  all  affect  the  relative 
rankings  of  ENM-materiel  pairs.  Table  S2  in  SI  provides 
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Table  1  List  of  ENM  physicochemical  characteristics  used  in  this  analysis 
Category  ENM  characteristic  Definition 


Fate 


Reactivity 


Structural 


Chemistry 


Application- 

specific 


Degradation 

potential 

Dispersibility 

Koc 

K-ow 

Persistence 

Bioaccumulation 

Half-life 

Toxicity 


Surface  charge/zeta 
potential 


Surface  reactivity 

Radical  formation 
Catalytic  reaction 
Flammability 
Explosivity 
Particle  size 


Density 
Composition 
Surface  area 
Molecular  structure 
Porosity 
Crystallinity 
Dustiness 
Solubility 

Aggregation 

Surface  chemistry 

Form 


Shape 


The  ability  of  the  ENM  to  be  decomposed/dissolved  by  bacteria  or  other  biological  means. 
Representative  of  “Fate”  due  to  its  ability  to  cover  other  ENM  characteristics  in  “Fate”  and  lack  of 
available  data  for  other  characteristics  in  this  category 

The  ability  of  the  ENM  to  become  distributed  continuously  throughout  a  particular  medium  (in  this  case 
air) 

Organic  carbon-water  partitioning  coefficient;  represents  the  ability  of  the  ENM  to  adsorb  to  organic 
carbon  and  move  through  carbon-rich  environments 

Octanol- water  partitioning  coefficient;  represents  the  tendency  to  sorb  to  non-polar,  lipophilic 
compounds  (lower  Kow  -  >  more  soluble  -  >  less  sorbing  to  solids) 

The  ability  of  the  ENM  to  remain  in  the  environment  without  modification  (e.g.,  biodegradation) 

The  ability  of  the  ENM  to  accumulate  in  the  tissues  of  organisms 

The  amount  of  time  it  takes  for  ENM  activity  to  be  reduced  by  half  (or  concentration  reduced  by  half) 

The  potential  for  adverse  health  effects,  if  exposed  to  a  given  ENM.  Representative  of  “Reactivity”  due 
to  its  ability  to  cover  other  ENM  characteristics  in  “Reactivity”  and  importance  to  understanding  its 
behavior  with  biological  systems 

The  electric  charge  present  at  the  interface  between  two  particles.  Zeta  potential  represents  the  degree  of 
repulsion  between  adjacent,  similarly  charged  particles.  Larger  absolute  zeta  potentials  lead  to  more 
stability  in  solutions.  Representative  of  “Reactivity”  due  to  its  importance  to  understanding  its 
behavior  with  biological  systems  particularly  relevant  for  ENMs 

The  ability  of  the  ENM  surface  to  interact  with  other  particles.  Generally  a  function  of  other  properties 
such  as  surface  charge/zeta  potential,  shape,  surface  area,  and  surface  chemistry 

The  ability  of  ENM  to  cause  a  release  of  free  radicals 
The  ability  of  ENM  to  enable  or  enhance  reactions 

The  ease  at  which  ENM  could  ignite  or  bum,  thereby  causing  fires  or  combustion  reactions  during  use 
The  tendency  of  ENM  to  explode 

The  typical  (modal)  diameter  for  a  distribution  of  raw  ENM  (often  there  is  a  size  distribution,  rather  than 
one  distinct  size).  Representative  of  “Structural”  due  to  the  relevance  for  ENMs  and  more  often 
reported  in  the  literature  compared  to  some  of  the  other  ENM  characteristics  in  this  category 

The  mass  of  ENM  per  unit  volume 

The  chemical  constituents  that  make  up  the  ENM 

The  surface  area  of  the  ENM  in  square  meters  (or  nanometers) 

The  molecular  formula  of  the  ENM,  including  the  bond  structure  and  strength 
A  measure  of  the  ‘empty’  spaces  in  a  materiel  as  volume 

The  degree  of  ordered  structures  in  the  materiel  (closely  related  to  porosity  and  molecular  structure) 
The  likelihood  of  ENM  becoming  airborne  during  use  or  disturbance 

The  ability  of  the  ENM  to  dissolve  in  a  solid,  liquid,  or  gaseous  solvent.  Representative  of  “Chemistry” 
due  to  its  importance  in  chemical  interactions  as  well  as  exposure  potential  with  biological  systems 

The  ability  of  individual  ENM  (nanoparticle)  to  physically/chemically  combine  into  larger  particles 
(related  to  agglomeration/flocculation) 

The  properties  that  affect  chemical  interactions  between  two  surfaces.  In  this  instance,  the  presence  and 
composition  of  surface  coatings  on  the  ENM 

The  physical  configuration  of  the  ENM  at  their  final  use  (e.g.,  particles,  bulk,  sheets,  fibers,  etc.). 
Representative  of  “Application-Specific”  due  to  its  importance  in  determining  exposure  and  hazard 
potentials,  in  which  different  forms  may  produce  different  responses  for  certain  ENMs 

The  normal  physical  appearance  of  the  raw  ENM  (e.g.,  tubes  and  spheres).  Representative  of 
“Application-Specific”  due  to  its  importance  in  determining  exposure  and  hazard  potentials,  in  which 
different  shapes  may  produce  different  responses  for  certain  ENMs 


Note  that  the  ENM  characteristics  shown  in  bold  below  were  those  chosen  as  being  representative  of  each  category,  a  designation  that  was  useful 
for  data  collection  and  risk  scoring  purposes 


additional  details  on  how  the  27  ENM  characteristics  relate 
to  each  of  these  three  risk  factors.  Other  authors,  such  as  Epa 
et  al.  2012,  Winkler  et  al.  2013,  and  Meesters  et  al.  2014, 


provide  more  details  on  how  ENM  physicochemical 
parameters  (characteristics)  relate  to  toxicological  effects 
and  exposure  pathways. 
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2.3  Identification  of  Army  materiel  characteristics 

As  the  relative  health  risks  considered  in  this  case  study 
pertain  to  ENMs  used  in  specific  Army  materiel  applica¬ 
tions,  Army  materiel  characteristics  were  also  identified 
that  may  play  a  significant  role  in  determining  the  relative 
risk  of  the  pairs  of  ENMs  and  the  materiel  in  which  they 
are  embedded.  Overall,  ten  Army  materiel  characteristics 
were  defined  using  data  from  the  Army  (Table  2).  Seven  of 
the  materiel  characteristics  were  used  in  scoring,  relevant 
for  risk  ranking  purposes,  while  three  characteristics  were 
only  used  for  informational  purposes. 

2.4  Data  acquisition  and  expert  elicitation  for  ENMs 
and  Army  materiel  characteristics 

To  streamline  the  data  acquisition  process  as  well  as  handle 
the  numerous  data  gaps  associated  with  information  on  each  of 
the  27  ENM  characteristics,  a  total  of  seven  ENM  character¬ 
istics  were  selected  as  being  representative  of  each  ENM 
category.  These  representative  ENM  characteristics  are 
shown  in  bold  in  Table  1  and  include  degradation  potential, 
surface  charge/zeta  potential,  toxicity,  particle  size,  solubility, 


form,  and  shape.  Quantitative  values  (e.g.,  solubility)  or 
qualitative  information  (e.g.,  form)  was  obtained  for  these 
seven  representative  characteristics  using  available  Army- 
supplied  or  literature-based  data.  These  values  were  subse¬ 
quently  incorporated  in  the  scoring  process  to  produce  semi- 
quantitative  scores  for  each  ENM  (see  subsequent  section  as 
well  as  Table  S2  in  SI  for  details)  and  served  as  surrogate  data 
for  the  other  characteristics  in  each  respective  ENM  charac¬ 
teristic  category.  For  example,  data  or  information  collected 
either  from  the  Army  or  the  literature  pertaining  to  the  deg¬ 
radation  potential  of  a  particular  ENM  would  be  used  as 
representative  data  (denoted  as  “representative”  in  the  char¬ 
acteristic  scoring  basis,  see  Table  S2  in  SI)  for  other  ENM 
characteristics  in  the  “fate”  ENM  characteristics  category 
such  as,  e.g.,  dispersibility  and  Koc  (denoted  as  “default”  in 
the  characteristic  scoring  basis,  Table  S2).  This  was  due  pri¬ 
marily  to  the  challenges  of  obtaining  data  for  all  27  ENM 
characteristics  and  therefore  served  as  a  way  to  manage  the 
data  gaps  in  the  data  collection  process.  As  noted  in  sub¬ 
sequent  sections,  there  is  also  a  built-in  functionality  to  the 
database  to  add  and  update  additional  information  and  data  for 
the  ENM  characteristics,  including  the  non-representative 
ENM  characteristics,  as  information  becomes  available. 


Table  2  List  of  Army  materiel  characteristics  used  in  this  analysis 

Materiel  Definition 

characteristic 


Amount 

Number  end  items 

Number  people 
exposed 

Acquisition  phase 


Use  patterns 


Incompatibility 

Method  of 
incorporation 

Toxicity  clearance3 
MSDSa 
Health  hazard 
assessment3 


The  amount  (%)  of  ENM  incorporated  into  the  materiel  (relates  to  release  potential,  exposure  potential)  (e.g.,  a  materiel 
containing  a  very  small  %  of  ENM  would  be  less  likely  to  release  the  ENM  and  would  result  in  a  smaller  exposure 
concentration) 

The  total  number  of  individual  final  (produced)  items  for  a  particular  ENM-materiel  pair  (relates  to  exposure  potential), 
e.g.,  if  5,000  end  items  are  produced,  the  likelihood  of  exposure  is  greater  than  a  materiel  with  currently  only  two  end 
items 

The  total  number  of  current  individuals  with  the  potential  for  exposure  to  the  ENM-containing  materiel  (relates  to 
exposure  potential),  e.g.,  if  three  people  have  the  potential  for  exposure  due  to  current  use,  rather  than  thousands,  then 
exposure  potential  is  considered  low 

The  current  status  of  the  ENM-containing  materiel  based  on  life  cycle  stage,  from  concept  design  to  production  and 
deployment  (relates  to  exposure  potential)  [e.g.,  a  materiel  that  is  still  in  the  concept  design  phase  (e.g.,  planning  only) 
would  have  no  exposure  potential,  whereas  a  materiel  that  has  been  deployed  for  use  could  potentially  have  a  large 
exposure  potential] 

A  descriptor  for  who  will  primarily  be  using  the  ENM-containing  materiel  in  its  current  stage  and  in  what  setting  (relates 
to  release,  exposure  potential,  and  toxicity  potential),  e.g.,  an  ENM  used  in  an  obscurant  would  theoretically  have  a 
higher  release,  exposure,  and  toxicity  potential  than  an  ENM  used  in  body  armor 

A  list  of  substances  that  may  be  incompatible  with  the  ENM-containing  materiel 

A  descriptor  for  how  the  ENM  is  incorporated  into  the  materiel  (i.e.,  on  the  surface,  in  a  polymer  matrix,  in  a  powder, 
etc.)  relates  to  release,  exposure,  and  toxicity  potential,  e.g.,  if  the  ENM  is  present  in  a  polymer  matrix,  then  the 
likelihood  of  release  and  subsequent  exposure/toxicity  would  be  diminished 

Yes/no  answer  on  whether  or  not  a  toxicity  clearance  has  been  performed  for  the  materiel  application  containing  ENMs 

Yes/no  answer  representing  the  presence/absence  of  a  material  safety  data  sheet  for  the  ENM  used  in  the  application 

Yes/no  answer  on  whether  or  not  a  health  hazard  assessment  has  been  performed  on  the  materiel  application  containing 
the  ENMs 


Note  that  these  three  Army  materiel  characteristics  were  not  used  in  risk  ranking  but  served  as  additional  descriptive  information 
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A  hierarchical  approach  was  used  to  acquire  data  for 
each  of  the  seven  representative  ENM  characteristics.  First, 
the  Army  supplied  the  first  round  of  data  attributed  to  the 
ENMs  and  Army  materiel,  and  then,  publically  available 
literature  was  subsequently  used  to  help  fill  in  the 
remaining  data  gaps.  Throughout  the  data  collection  per¬ 
iod,  the  Army  provided  data  and  information  for  only  39  % 
of  the  ENM-materiel  pairs.  Therefore,  the  literature  in  the 
form  of  Material  Safety  Data  Sheets  (MSDSs)  as  well  as 
scientific  journals  was  accessed  subsequently.  A  MSDS 
typically  provides  some  information  on  the  ENM  or  bulk 
materials,  e.g.,  particle  size,  shape,  solubility,  and  toxicity, 
although  there  was  significant  variability  in  the  type  and 
quantity  of  information  provided  between  MSDSs.  Fur¬ 
thermore,  follow-up  communication  with  the  ENM  man¬ 
ufacturer  used  by  Army  personnel  was  also  conducted  in 
some  instances  in  order  to  obtain  as  much  information  as 
possible  from  an  MSDS.  Then,  for  information  that  could 
not  be  found  using  MSDSs,  existing  scientific  literature 
was  used  to  ascertain  necessary  data  or  descriptive  infor¬ 
mation  on  toxicity  and  surface  charge/zeta  potentials. 
Several  literature  values  were  examined  to  account  for 
some  of  the  variability  in  ENMs  used  across  different 
studies.  If  ENM- specific  data  could  not  be  found,  bulk- 
scale  characteristics  of  the  primary  constituent  were  col¬ 
lected  from  a  corresponding  MSDS  to  serve  as  a  place¬ 
holder  until  ENM-specific  data  become  available.  Finally, 
for  the  data  gaps  that  could  not  be  addressed  due  to  a  lack 
of  scientific  literature,  existing  chemical  databases  includ¬ 
ing  EPA  EpiSuite  (USEPA  2014)  and  ChemSpider  (Pence 
and  Williams  2010)  were  used  to  obtain  initial  values, 
often  based  on  information  related  to  bulk- scale  materials. 

These  data  and  information  were  collected  and  incorpo¬ 
rated  into  the  underlying  database  of  the  risk  ranking  tool, 
Tool  for  ENM  Application  pair  Risk  Ranking  (TEARR). 
TEARR  was  designed  with  Microsoft  Office  2007  with 
Visual  Basic  6.5  code  and  implemented  using  an  access 
database  with  a  corresponding  user  interface  (see  Figures 
S1-S4  in  SI  for  screenshots).  The  underlying  database  in 
TEARR  therefore  served  as  the  source  of  underlying  data 
used  for  an  expert  elicitation  of  the  ENM  risk  factor  scores 
for  each  representative  ENM  characteristic  as  well  as  Army 
materiel  impact  scores  as  explained  below. 

In  an  expert  elicitation  process,  two  prominent  nano¬ 
toxicologists  were  used  to  elicit  scores  (i.e.,  1,3,  and  5)  for 
the  three  ENM  risk  factors  (release  potential,  exposure 
potential,  and  toxicity  potential)  for  each  of  the  represen¬ 
tative  ENM  characteristics  based  on  the  data  collected  (see 
SI  for  more  details).  To  assign  a  score  to  a  particular  ENM 
risk  factor  related  to  a  given  ENM-materiel  pair,  the  fol¬ 
lowing  question  was  posed:  “For  each  inherent  ENM 
characteristic  associated  with  a  given  ENM-materiel  pair, 
based  on  a  given  value  for  the  ENM  characteristic,  what  is 


the  release  potential,  exposure  potential,  and/or  toxicity 
potential  of  the  ENM-materiel  pair?”  An  ENM  risk  factor 
score  of  1  relates  to  a  low  potential,  3  medium  potential, 
and  5  high  potential.  The  ranking  scale  of  1,  3,  and  5  was 
chosen  based  on  its  use  in  other  risk  ranking  analyses  for 
ENMs,  such  as  Hock  et  al.  (2008),  and  was  deemed  a 
simple  approach  to  qualitatively  assign  low-,  medium-,  and 
high-risk  potentials.  The  individual  ENM  risk  score  rep¬ 
resents  the  relative  risk  that  a  particular  ENM  characteristic 
will  have  on  each  ENM  risk  factor.  For  the  other  ENM 
(non-representative)  characteristics,  scoring  rules  were 
used  to  provide  the  scores  for  the  risk  factors  (see  Table  S2 
in  SI),  whereby  default  scores  were  assigned  to  the 
remaining  the  ENM  characteristics  in  each  ENM  charac¬ 
teristic  category.  See  Section  4  in  SI  for  full  details  of  the 
expert  elicitation  process. 

Similar  to  eliciting  scores  of  ENM  risk  factors,  expert 
elicitation  was  also  used  to  assign  Army  materiel  impact 
scores  for  each  Army  materiel  characteristic.  An  Army 
materiel  impact  score  represents  the  impact  that  a  partic¬ 
ular  materiel  characteristic  will  have  on  the  total  risk  of  an 
ENM-material  application  pair,  and  values  of  0,  0.5,  1,  2, 
5,  10,  or  100  were  used.  A  materiel  impact  score  of  0 
indicates  that  ENM  used  in  this  Army  materiel  is  expected 
to  negate  the  overall  risk  of  the  application  pair  for  a 
specific  receptor/exposure  (e.g.,  for  soldiers  wearing  body 
armor,  it  is  expected  that  there  is  no  risk  because  release  of 
the  ENM  is  not  expected  to  occur  given  the  Method  of 
Incorporation).  An  impact  score  of  0.5  indicates  that  the 
ENM  used  in  the  Army  materiel  is  expected  to  decrease  the 
overall  risk  of  the  ENM-materiel  pair  for  a  specific  sce¬ 
nario  by  a  factor  equal  to  the  impact  score,  although  the 
ENM  is  still  considered  to  pose  some  risk  related  to 
potential  release  into  the  environment  and  exposing 
receptor  populations.  Impact  score  of  1  indicates  that  ENM 
used  in  the  Army  materiel  may  or  may  not  influence  the 
overall  risk  of  the  ENM-materiel  pair  for  a  specific  sce¬ 
nario,  or  no  data/information  (i.e.,  no  information  supplied 
by  Army  for  materiel).  Impact  score  >1  (i.e.,  2,  5,  10,  or 
100)  indicates  that  the  ENM  used  in  the  Army  materiel  is 
expected  to  increase  the  overall  risk  of  the  ENM-materiel 
pair  by  a  factor  equal  to  or  greater  than  the  materiel  impact 
score  (e.g.,  if  the  number  of  people  potentially  exposed  is 
in  the  thousands,  this  would  be  considered  substantial  rel¬ 
ative  to  the  current  Army-supplied  data;  thus,  the  materiel 
impact  score  for  the  ‘number  of  people  exposed’  materiel 
characteristic  would  be  >1).  Average  Army  materiel 
impact  scores  were  derived  by  averaging  the  materiel 
impact  scores  for  all  Army  materiel  characteristics  and 
indirectly  relate  to  the  release,  exposure,  and  toxicity 
potential  of  a  particular  application. 

After  ENM  risk  factors  were  assigned  through  the  expert 
elicitation  process,  ENM  risk  factor  scores  were  weighted 
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by  the  experts  in  order  to  increase  or  decrease  the  relative 
importance  of  certain  ENM  characteristics  which  may  be 
more  uncertain  or  of  lesser  importance  compared  with 
others.  Because  there  is  still  substantial  uncertainty  to 
understanding  the  potential  release,  exposure,  and  toxicity 
potential  of  ENMs  in  various  applications,  the  user  has  the 
ability  to  adjust  the  ENM  weights  in  the  risk  ranking 
algorithm  (wl,  Eq.  1).  This  weighting  scheme  also  has  the 
advantage  of  providing  valuable  information  on  the  sensi¬ 
tivity  of  the  ranking  results,  whereby  the  user  can  compare 
the  ranking  results  when  using  different  weights  for  ENM 
risk  factor  scores.  Similar  to  assigning  ENM  risk  scores,  a 
binning  system  was  used  (i.e.,  0,  1,2)  to  assign  a  weight  to 
each  ENM  characteristic  by  the  experts  based  on  the  fol¬ 
lowing  question:  “How  significant  to  the  overall  relative 
riskiness  of  the  ENM-materiel  pair  is  the  release  potential, 
exposure  potential,  and/or  toxicity  potential  due  to  this 
ENM  characteristic?”  A  weight  of  0  relates  to  not  signif¬ 
icant,  1  moderately  or  equally  significant  to  other  ENM 
risk  factor  scores,  and  2  very  or  more  significant  than  other 
ENM  risk  factor  scores.  A  default  weight  of  1  was  used  in 
the  baseline  set  of  results.  Average  ENM  risk  factor  scores 
were  then  determined  by  first  multiplying  each  ENM  risk 
factor  score  by  its  corresponding  weight,  then  summing  the 
weighted  score  for  each  characteristic  chosen  by  the  user, 
and  dividing  by  the  total  number  of  characteristics  chosen. 

Several  assumptions  were  made  to  relate  the  three  ENM 
risk  factors  to  the  overall  risk  of  an  ENM-Army  materiel 
pair.  First,  it  was  assumed  that  the  ENM-materiel  pair 
cannot  pose  a  risk  to  a  receptor  if  exposure  does  not  occur. 
Similarly,  it  was  assumed  that  exposure  is  not  likely  if  the 
ENM  is  not  released  into  an  environment  from  the  Army 
materiel.  It  was  also  assumed  that  a  direct  toxicity  char¬ 
acteristic  for  the  Army  materiel  would  thereby  increase  the 
toxicity  scoring  for  an  individual  ENM.  For  this,  the  use  of 
Army  materiel  characteristics,  such  as  Method  of  Incor¬ 
poration  and  Use  Patterns,  appropriately  will  allow  the 
adjustment  of  the  overall  toxicity  of  the  Army  materiel  and 
subsequent  relative  risk  score  by  considering  how  the  ENM 
is  being  used  in  the  materiel  (e.g.,  Can  the  ENM  be 
released  from  the  materiel?  Can  the  receptor  subsequently 
be  exposed  to  the  ENM  and  can  health  effects  then  occur 
from  exposure?).  Finally,  potential  exposures  through 
waste  or  other  end-of-life  cycle  stages  were  not  included  in 
the  present  study. 

2.5  Relative  risk  ranking  parameters  and  ranking 
algorithm 

After  collecting  data  and  information  on  ENMs,  Army 
materiel,  and  eliciting  ENM  risk  factor  and  Army  materiel 
impact  scores,  this  information  was  incorporated  into  a 
relative  risk  ranking  algorithm  within  TEARR  (Eq.  1).  The 
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outcome  of  the  risk  ranking  algorithm  produces  relative 
risk  results  that  identify  the  higher  risk  ENMs,  Army 
materiel,  and  ENM-materiel  pairs  according  to  receptor 
type,  release  type,  and  exposure  route  based  on  the  scenario 
selected  by  the  user.  Overall,  the  relative  risk  of  a  partic¬ 
ular  ENM-materiel  pair  is  calculated  based  on  the  average 
ENM  risk  score  multiplied  by  the  average  Army  materiel 
impact  score. 

The  relative  risk  score,  R ,  for  a  user  profile,  /,  and 
release  profile,  j,  is  calculated  in  Eq.  1  as  follows  for  a 
specific  ENM-Army  materiel  pair.  An  example  calculation 
using  Eq.  1  can  also  be  found  in  Supplementary  Informa¬ 
tion  (Section  5). 
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where:/*  =  [dermal,  ingestion,  inhalation];/  =  [civilian 
worker,  soldier]  \j  —  [occupational,  accidental]  ;m  =  [1,  2, 
or  3]  and  corresponds  to  the  risk  factors  (release  potential, 
exposure  potential,  or  toxicity  potential)  for  a  particular 
ENM  characteristic,  kl;**l  =  the  total  number  of  ENM 
characteristics  ranked  by  the  user;RS  =  [1,3,  or  5]  and  is 
the  relative  risk  score  for  a  particular  ENM  characteristic, 
kl,  and  risk  factor,  m;£l  =  ENM  characteristic  wl  =  [0,  1, 
or  2]  and  corresponds  to  the  ranking  weight  assigned  by  the 
user;**2  =  the  total  number  of  Army  materiel  characteris- 
tics;IS  =  [0,  0.5,  1,  2,  5,  10,  or  100]  and  corresponds  to  the 
user- weighted  materiel  impact  score  for  a  particular  Army 
materiel  characteristic,  k2;£2  =  Army  materiel 
characteristic 


2.6  Running  scenarios  and  viewing  results 

In  order  to  generate  the  baseline  results  of  the  data  and 
information  contained  within  TEARR,  the  user  first  creates 
the  scenario  of  interest.  In  this  step,  the  user  first  selects  the 
receptor  (i.e.,  soldier,  worker),  release  type  (i.e.,  accidental, 
operational),  and  exposure  route  (i.e.,  dermal,  ingestion, 
inhalation)  of  interest  (Figure  SI  in  SI  shows  a  screenshot 
of  this  setup  screen  in  TEARR).  Next,  the  user  selects  the 
Army  materiel  and  ENMs  (see  Table  SI  in  SI  for  a  com¬ 
plete  list  of  ENMs  and  materiel).  Note  that  only  the  ENMs 
included  in  the  selected  Army  materiel  will  be  available  for 
selection  in  TEARR,  i.e.,  TEARR  relies  on  specific  ENM- 
materiel  pairs.  The  user  also  selects  the  ENM  characteris¬ 
tics  of  interest  (see  Table  1).  Next,  the  user  is  able  to  view 
and  adjust  the  Army  materiel  impact  scores  (default  value 
is  set  to  1  if  no  prior  impact  score  has  been  entered  pre¬ 
viously,  and  a  default  weight  of  1  was  used  in  the  baseline 
set  of  results;  see  Figure  S2  in  SI)  as  well  as  ENM  scores 
and  weights  (see  Figure  S3  in  SI).  Then,  the  user  runs  the 
scenario  in  order  to  generate  the  relative  risk  ranking  scores 
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and  output.  On  this  results  screen  (not  shown  for  confi¬ 
dentiality  reasons;  i.e.,  Army- sensitive  related  informa¬ 
tion),  the  user  is  able  to  view  the  ENM  risk  scores,  Army 
materiel  impact  scores,  relative  risk  score,  and  relative  risk 
rank  for  the  ENM-materiel  pairs.  In  essence,  the  results 
from  TEARR  show  not  only  a  ranked  list  of  the  ENM- 
materiel  pairs  but  also  results  based  solely  on  the  ENM  or 
Army  materiel  type.  Finally,  the  summary  results  screen 
(see  Figure  S4  in  SI)  provides  an  overview  of  the  relevant 
receptor,  release  and  exposure  types,  average  Army  mate¬ 
riel  impact  scores,  average  ENM  risk  score,  as  well  as 
overall  relative  risk  scores  listed  in  descending  order. 

In  addition  to  viewing  the  relative  risk  ranking  results  of 
ENMs,  Army  materiel,  and  ENM-materiel  pairs,  the  user  is 
also  able  to  view  the  data  sources  used  to  determine  the 
ENM  risk  scores  and  Army  materiel  impact  scores  (i.e., 
Army  supplied,  literature  based,  expert  elicitation,  and 
default  scoring  based  on  scoring  rules  shown  in  Table  S2) 
on  a  separate  screen  within  TEARR  (Figure  S5  in  SI).  The 
user  is  also  able  to  view  supplemental  information  which 
was  provided  by  the  Army  but  not  incorporated  into  the 
TEARR  algorithm  in  order  to  view  any  additional  infor¬ 
mation  related  to  Army  materiel  characteristics  that  may  be 
relevant  to  the  selected  scenario.  This  is  listed  under  the 
“supplemental  info”  table  (not  shown  here). 

2.7  Updating  data  and  information  within  TEARR 

TEARR  has  the  ability  to  evolve  and  adapt  to  the  incor¬ 
poration  of  new  information  as  well  as  expert  knowledge. 
Users  are  currently  able  to  modify  the  Army  material 
impact  scores  as  well  as  ENM  risk  factor  scores  and 
weights  if  she/he  is  aware  of  additional  information  not 
currently  included  in  the  underlying  database  within 
TEARR.  TEARR  also  has  the  ability  for  users  to  modify 
the  default  risk  scores  used  in  the  underlying  scoring 
scheme  as  more  information  and  data  become  available. 


3  Results  and  discussion 

3.1  ENM  and  Army  materiel  inventory 

A  total  of  45  ENMs,  30  Army  materiel  applications,  and 
133  separate  ENM- Army  materiel  pairs  were  identified  in 
this  analysis.  The  ENMs  were  grouped  into  seven  catego¬ 
ries  based  on  their  composition:  carbon-based,  inorganic, 
metals,  metal  oxides,  quantum  dots,  other,  and  unknown 
(see  Table  SI  in  SI).  The  Army  materiel  was  grouped  into 
six  categories  based  on  their  functionality  and  use:  chem¬ 
icals,  electronics,  equipment,  munitions,  support,  and 
structural  material  (Table  SI  in  SI).  Figure  1  shows  the 
distribution  of  ENMs  across  the  ENM-materiel  pairs 


( N  =  133),  revealing  that  carbon  nanotubes  (CNT)  make 
up  the  largest  number  of  Army  materiel  applications 
(A  =  13),  followed  by  silver  (Ag),  gold  (Au),  aluminum 
(Al),  and  alumina  (A1203)  (all  with  N  =  6).  While  a  full 
list  of  ENMs  and  Army  materiel  is  listed  in  Table  SI  of 
Supplementary  Information  (SI),  a  complete  list  of  all 
ENM-Army  materiel  pairs  is  not  provided  in  this  analysis 
due  to  sensitive  nature  of  this  information. 

3.2  Relative  risk  scores 

Given  the  various  options  that  users  may  select  when  set¬ 
ting  up  a  given  scenario  (i.e.,  receptors  types,  release  types, 
three  exposure  routes,  ENMs,  and  materiel),  there  were 
1,596  different  scenarios  possible  for  ranking  in  TEARR. 
Based  on  the  risk  ranking  algorithm,  the  relative  risk  scores 
can  potentially  range  from  0  to  3,000,  with  3,000  repre¬ 
senting  the  ENM-materiel  pair  with  the  highest  relative 
risk  score.  To  place  this  within  context,  the  components  of 
the  relative  risk  score  include  the  average  ENM  risk  score 
(which  can  range  1-30)  and  the  average  Army  materiel 
impact  score  (which  can  range  0-100).  Figure  2  illustrates 
the  final  relative  risk  scores  for  all  1,596  scenarios  along 
with  each  ENM  risk  score  and  Army  materiel  impact  score. 

Although  there  is  the  possibility  for  a  risk  score  to  reach 
3,000,  the  highest  achieved  score  based  on  current 
knowledge  for  the  ENMs  and  Army  materiel  in  TEARR 
was  42.84  (Table  3).  This  is  most  likely  due  to  the  fact  that 
the  baseline  Army  materiel  impact  scores  did  not  include 
the  highest  potential  scores  (e.g.,  10,  100)  for  most  ENM- 
materiel  pairs  given  the  high  degree  of  uncertainty 
regarding  the  maximum  values  possible  for  a  given  Army 
materiel  characteristic.  To  investigate  this  further,  the 
values  of  10  and  100  were  excluded  from  the  Army 
materiel  impact  scores  in  a  separate  analysis,  which 
resulted  in  the  overall  maximum  final  risk  score  adjusted  to 
150.  Therefore,  the  likely  cause  of  the  relatively  low 
maximum  risk  score  compared  with  a  theoretical  maximum 
is  due  to  the  relatively  low  Army  materiel  impact  scores 
used  in  the  TEARR  baseline  analysis. 

3.3  Risk  ranking  results 

Baseline  results  from  TEARR  showed  the  relative  rankings 
of  Army  materiel,  ENMs,  as  well  as  ENM-Army  materiel 
pairs.  Looking  at  the  individual  ENM-materiel  pairs,  the 
baseline  risk  ranking  results  show  that  aluminum  (Al), 
copper  (Cu),  and  titanium  (Ti)  flakes  used  in  smokes  and 
obscurants  ranked  the  highest  on  the  risk  scale  for  sce¬ 
narios  primarily  involving  accidental  civilian  worker  and/ 
or  soldier  inhalation  (data  not  shown  for  confidentiality 
reasons).  Due  to  the  anticipated  number  of  end  items  and 
potential  for  many  people  to  be  exposed,  the  dermal  and 
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Fig.  1  Distribution  of  nanomaterials  based  on  the  current  inventory  of  133  unique  materiel  applications 
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ENM-Materiel  Application  Scenario 

Fig.  2  Baseline  relative  risks  scores  of  all  possible  scenarios  currently  in  TEARR  (N  =  1,596) 


Table  3  Summary  of  relative  risk  scores  for  all  ENM-Army  materiel  scenarios  in  the  baseline  analysis 


Scoring  component 

Minimum 

Maximum3 

Mean 

Variance 

ENM  risk  factor  score 

1.00 

15.68/30 

7.80 

8.80 

Army  materiel  impact  score 

0.00 

3.29/100 

1.06 

0.08 

Final  risk  score 

0.00 

42.84/3,000 

8.20 

17.73 

a  These  scores  represent  the  final  ENM  risk  factor  and  materiel  impact  scores  (out  of  the  maximum  scores  possible)  which  are  based  on  weighted 
averages  across  all  ENM  characteristics  or  all  materiel  characteristics,  respectively 


ingestion  exposure  routes  also  ranked  high  for  these  Army 
materiel,  based  on  current  knowledge.  Additionally,  silver 
(Ag)  ranked  relatively  high  in  the  relative  risk  scoring  for 
accidental  ingestion  of  Ag-containing  Army  materiel  used 
in  energy  and  sensor  applications,  and  carbon  nanotubes 
(CNT)  also  ranked  relatively  high  based  on  worker  inha¬ 
lation  from  CNTs  used  in  research  and  development 


practices.  In  contrast,  the  lowest  scoring  ENM-materiel 
pair  scenarios  (excluding  scenarios  involving  unknown 
substances)  included  aluminums,  ceramics,  and  carbon- 
containing  ENMs  used  in  solid  matrices  (such  as  armor, 
vehicles,  and  personal  protection  equipment),  because  the 
likelihood  of  these  ENMs  being  released  during  opera¬ 
tional  (or  even  accidental)  use  was  very  low,  or  because 
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many  of  the  low  scoring  items  had  very  few  end-use  items 
or  were  still  in  the  development  phase. 

Across  all  Army  materiel  categories  based  on  the 
average  final  relative  risk  score,  the  ‘smokes  and  obscu¬ 
rants’  category  had  the  highest  potential  risk  based  on 
currently  available  information,  with  an  average  score  of 
19.02  (Fig.  3).  Smokes  and  obscurants  incorporate  ENMs 
that  were  also  shown  to  be  more  highly  ranked,  such  as 
aluminum  (Al),  copper  (Cu),  brass,  and  titanium  dioxide 
(Ti02)  (results  not  shown).  The  next  highest  risk  Army 
materiel  categories  include  coolants  (average  risk  score  of 
11.67)  and  greases  (average  risk  score  of  10.67),  containing 
ENMs  such  as  alumina  (A1203)  and  CNTs.  The  categories 
with  the  overall  lowest  risk  included  rations  (3.12),  pro¬ 
jectiles  (2.42),  and  communication  (2.42)  with  largely 
unknown  ENM  composition.  The  average  risk  scores  for 
this  materiel  were  relatively  low,  most  likely  due  to  their 
incorporation  into  solid  matrices  or  other  materiel  expected 
to  have  low  exposure  potentials.  Several  categories  also 
showed  zero  risk  (not  shown),  predominantly  due  to  a  lack 
of  data  or  information  regarding  the  ENMs  being  used  in 
those  applications. 

The  five  highest  ranked  ENMs  included  in  TEARR  were 
as  follows:  copper  (Cu)  (average  risk  score  of  22.9),  brass 
(15.95),  titanium  dioxide  (Ti02)  (14.41),  palladium  (Pd) 
(11.46),  and  silver  (Ag)  (10.42)  (Fig.  4).  These  ENMs  were 
incorporated  into  smokes  and  obscurants,  sensors,  energy, 


and  imaging  (results  not  shown  due  to  confidentiality  rea¬ 
sons;  Army- sensitive  information).  In  contrast,  the  ENMs 
that  were  ranked  the  lowest  in  TEARR  were  as  follows: 
silicon  (Si)  (4.96),  carbon  aluminum  composite  (CarbAl) 
(4.79),  ceramics  (4.65),  miscellaneous  (4.32),  and  unknown 
ENMs  (2.74).  These  ENMs  were  used  in  a  wide  range  of 
Army  materiel:  research  and  development,  lubricants, 
energy,  batteries,  protection,  energetics,  coatings  and 
paintings,  air  filtration  and  purification,  rations,  and  com¬ 
munication.  The  average  risk  scores  for  these  ENMs  were 
most  likely  due  to  the  low  expected  exposure  potential  and 
a  lack  of  data  or  information  regarding  the  physicochemi¬ 
cal  parameters  of  these  ENMs. 

3.4  Data  sources,  uncertainty,  and  data  gaps 

The  relative  risk  ranking  approach  presented  here  repre¬ 
sents  a  mathematically  simple,  yet  comprehensive,  appli¬ 
cation  that  facilitates  the  use  of  available  data  and  expert 
judgment,  as  well  as  hypothesis  testing  (e.g.,  understanding 
the  impact  of  weighting  schemes).  Even  for  relatively 
simple  mathematical  constructs  such  as  this  ranking  tool, 
reviewing  the  data  sources  and  underlying  uncertainties  is 
critically  important  to  determine  what  types  of  research 
will  lead  to  substantial  improvements  in  the  quality  of  the 
risk  predictions  and  what  input  data  should  be  collected 
and  refined.  The  outcomes  from  the  TEARR  framework 


Fig.  3  Average  relative  risk 
scores  across  Army  materiel 
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Fig.  4  Average  relative  risk  scores  across  ENMs 

can  be  used  to  help  identify  these  data  gaps,  which  in  turn 
could  reduce  the  uncertainties  associated  with  risk  rank¬ 
ings,  resulting  in  improved  relative  risk  estimates.  It  should 
be  noted  that  the  focus  here  is  epistemic  uncertainty,  or 
uncertainty  that  may  be  reduced  from  further  research  as 
opposed  to  uncertainties  that  are  stochastic  in  nature 
(Grieger  et  al.  2009). 

In  TEARR,  a  review  of  the  data  sources  used  may  help 
compare  the  relative  importance  of  various  uncertainties 
that  may  be  associated  with  each  source  (Figure  S5  in  SI). 
The  data  sources  used  for  each  scenario  run  in  TEARR  are 
provided  as  a  percentage  of  all  data  sources  and  act  as  a 
final  and  critical  step  in  the  ranking  process,  especially 
given  the  extent  of  our  data  gaps  and  use  of  four  types  of 
data  [i.e.,  Army  supplied,  literature  derived,  expert  elici¬ 
tation,  or  user  applied  (default)].  In  this  analysis,  expert 
judgment  in  the  case  of  Army  materiel  impact  scoring  and 
default  scoring  in  the  case  of  ENM  risk  scores  was  pre¬ 
dominantly  the  basis  for  the  scoring  in  TEARR  (Figure  S5 
in  SI).  As  shown  in  Figure  S5  in  SI,  41  %  of  the  Army 
materiel  impact  scores  were  derived  from  Army- supplied 
data/information,  while  11  %  of  ENM  risk  scores  were 
based  on  literature  values,  8  %  on  expert  judgment,  and 
7  %  on  Army-supplied  data/information.  It  is  envisioned 
that  future  applications  and  modifications  to  TEARR  will 
include  an  updated  database  which  will  incorporate  new 
data  and  information  from  literature  sources  as  it  becomes 
available.  For  example,  the  ability  to  tap  into  publically 
accessible  databases  that  focus  on  the  environmental, 
health,  and  safety  (EHS)  implications  of  ENMs,  such  as  the 
Nanomaterial  Registry  (www.nanomaterialregistry.org), 
would  be  ideal  in  order  to  be  able  to  access  (possibly  in  real 


time)  the  most  current  data  set  on  ENMs  in  which  to  update 
the  TEARR  database. 

In  addition  to  a  review  of  the  underlying  data  sources 
used  in  TEARR,  the  relative  risk  scores  from  the  baseline 
analysis  appear  to  be  low  compared  with  the  maximum 
possible  risk  score  (i.e.,  42.84  out  of  3,000).  While  this 
could  indicate  that  very  few  ENM-materiel  pairs  pose  a 
high  risk,  a  closer  look  at  the  data  used  to  support  the 
baseline  analysis  suggests  that  our  current  lack  of  empirical 
data  led  to  lower  relative  risk  scores  in  order  to  reduce 
potential  biases  in  the  scoring  estimates.  In  other  words,  the 
prevalence  of  data  gaps  and  uncertainties  related  to  ENM 
and  Army  materiel  characteristics  may  have  erred  in  the 
production  of  false  negatives  rather  than  of  false  positives. 
Further  work  and  modifications  to  TEARR,  including  the 
incorporation  of  more  data  from  recently  published  studies 
and/or  access  to  publically  available  databases  focused  on 
EHS  implications  of  ENMs,  are  required  in  order  to  refine 
these  baseline  results  generated  thus  far. 

As  stated  previously  in  this  analysis,  the  selection  of  the 
seven  representative  ENM  characteristics  was  performed  in 
order  to  streamline  data  collection  efforts  as  well  as  man¬ 
age  the  data  gaps  related  to  obtaining  data  for  each  of  the 
ENM  characteristics  for  all  ENMs  in  this  analysis.  In  fact, 
obtaining  the  remaining  20  non-representative  ENM  char¬ 
acteristics  was  extremely  challenging,  as  data  gaps  were 
exceedingly  prevalent,  thus  leading  to  the  use  of  repre¬ 
sentative  ENM  characteristics  as  proxies  for  the  other 
ENM  characteristics  in  the  same  category.  Even  doing  this, 
the  seven  representative  ENM  characteristics  (size,  degra¬ 
dation  potential,  surface  charge/zeta  potential,  solubility, 
toxicity,  shape,  and  form)  contained  data  gaps  of  10,  25, 
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48,  21,  23,  32,  and  10  %,  respectively  (as  shown  in  the 
underlying  database,  not  shown  here).  While  expert  elici¬ 
tation  and  literature  reviews  were  conducted  to  reduce  data 
gaps  for  the  initial  evaluation,  further  data  collection  for 
these  characteristics,  particularly  toxicity  and  solubility,  as 
well  as  data  points  that  relied  on  bulk- scale  materials  rather 
than  ENM  specifically  are  still  needed.  Therefore,  the  ini¬ 
tial  relative  risk  ranking  based  on  the  representative  ENM 
characteristics  is  characterized  by  substantial  uncertainty 
due  to  the  use  of  default  scores  for  the  remaining  ENM 
characteristics.  Considering  default  ENM  characteristics,  a 
significant  reduction  in  the  ranking  uncertainty  (currently 
defined  as  source  information)  could  be  achieved  by 
obtaining  additional  ENM-specific  data  for  ENM-materiel 
pairs,  such  as  surface  chemistry,  chemical  composition, 
surface  reactivity,  and  partitioning  coefficients.  It  is  noted 
that  a  substantial  amount  of  ongoing  research  is  currently 
being  conducted  to  improve  test  methods  to  enhance 
characterization  efforts  of  ENMs,  e.g.,  Oomen  et  al. 
(2014),  and  hence,  further  refinements  and  revisions  to 
TEARR  will  need  to  incorporate  recently  published 
studies. 


4  Conclusions  and  recommendations 

This  analysis  documents  the  methodology  and  approach 
taken  to  develop  and  implement  a  relative  risk  ranking  tool 
for  ENMs  in  Army  materiel  applications.  The  output  from 
this  risk  ranking  tool,  TEARR,  provides  relative  risk  scores 
and  ranks  for  ENMs,  Army  materiel,  and  ENM-material 
pairs,  thereby  identifying  the  ENMs,  materiel,  and  ENM- 
materiel  pairs  which  were  ranked  the  highest  (and  lowest)  for 
further  analysis.  In  order  to  accomplish  this,  ENMs  and 
associated  Army  materiel  were  identified  and  inventoried 
based  on  collaboration  with  the  Army.  Then,  a  risk  ranking 
algorithm  was  developed  based  on  ENM  and  Army  materiel 
characteristics.  Finally,  the  TEARR  risk  ranking  algorithm 
was  implemented  and  tested  on  the  identified  ENMs,  mate¬ 
riel,  and  ENM-materiel  pairs,  producing  relative  risk  scores 
and  ranks  for  the  ENMs,  materiel,  and  pairs.  A  total  of  45 
ENMs,  30  Army  materiel,  and  133  ENM-materiel  pairs  were 
identified  in  the  analysis.  Across  all  ENM-materiel  pairs,  the 
maximum  relative  risk  score  generated  was  42.84  out  of  a 
possible  3,000 — relatively  low — most  likely  due  to  the 
prevalence  of  data  gaps  related  to  the  Army  materiel  impact 
scores.  Therefore,  future  modifications  of  TEARR  which 
incorporate  new  data  and  information  as  it  arises  are  clearly 
needed  in  order  to  refine  this  risk  ranking  tool. 

Across  all  ENM-materiel  pairs  included  in  TEARR, 
inhalation  from  accidental  exposures  to  CNTs  and  copper 
flakes  incorporated  into  energy  and  obscurant  materiel  by 
Army  researchers  ranked  the  highest  relative  to  the  other 


ENM-materiel  pairs  evaluated  in  this  baseline  assessment. 
Other  highly  ranked  ENM-materiel  pairs  included  acci¬ 
dental  ingestion  of  silver-containing  Army  materiel  used  in 
energy  and  sensor  applications  as  well  as  worker  inhalation 
of  CNTs  used  in  research  and  development  practices.  In 
addition  to  these  findings,  it  is  also  found  that  data  gaps  and 
uncertainties  were  extremely  prevalent  across  most,  if  not 
all,  identified  ENMs  and  Army  materiel.  In  order  to  handle 
this,  several  different  approaches  were  taken  including  the 
use  of  representative  ENM  characteristics  to  serve  as  proxies 
for  other  ENM  characteristics  in  the  same  category,  the  use 
of  expert  judgment  to  elicit  ENM  risk  scores  and  weights  as 
well  as  Army  materiel  impact  scores  and  weights.  Further¬ 
more,  TEARR  was  developed  in  order  to  allow  the  incor¬ 
poration  of  new  or  additional  data  as  it  becomes  available. 

The  main  advantages  of  TEARR  include  the  ability  to 
provide  valuable  information  related  to  the  potential  rela¬ 
tive  risks  associated  with  ENM-Army  materiel  pairs  based 
on  current  knowledge.  TEARR  also  incorporates  expert 
judgment  as  well  as  mechanisms  to  update  the  underlying 
baseline  database  as  more  knowledge  becomes  available. 
Additionally,  it  is  one  of  the  first  relative  risk  ranking  tools 
that  incorporate  both  the  unique  characteristics  of  the 
ENMs  in  conjunction  with  the  characteristics  of  the 
application  (i.e.,  Army  materiel  in  this  case)  and  thereby 
relevant  for  more  real-world  scenarios.  The  results  from 
TEARR  may  be  used  to  help  prioritize  additional  research, 
such  as  in-depth  risk  evaluations  or  further  nanotoxico- 
logical  research  pertaining  to  the  highest  ranked  ENMs, 
materiel,  or  ENM-materiel  pairs.  Finally,  the  fundamental 
methodology  and  risk  ranking  algorithm  developed  in 
TEARR  may  be  applicable  to  other  occupational  and 
environmental  settings  involving  ENMs  and  therefore 
easily  translated  to  other  application  scenarios.  Some  of  the 
major  shortcomings  of  TEARR  primarily  involve  the 
uncertainties  and  data  gaps  related  to  the  ENMs  and  Army 
materiel.  As  any  risk  ranking  or  analysis  tool  is  subject  to 
the  quality  of  the  data  stored  within,  so  is  the  TEARR 
database  and  risk  ranking  tool,  in  that  the  results  are  highly 
dependent  upon  the  quality  of  the  underlying  data.  It 
appears  that  the  relative  risk  scores  may  be  heavily  influ¬ 
enced  by  the  uncertainties  and  data  gaps  related  to  the 
Army  materiel  impact  scores,  producing  lower  than 
expected  risk  ranking  results.  Thus,  further  modifications  to 
TEARR  which  rely  on  updated  information  are  clearly 
needed.  Furthermore,  expert  elicitation  based  on  a  total  of 
two  nanotoxicology  and  ENM  risk  experts  was  used  as  a 
basis  to  score  the  ENM  risk  scores  and  weights  as  well  as 
Army  materiel  impact  scores  and  weights.  In  comparison,  a 
larger  expert  elicitation  panel  (i.e.,  6-10  experts)  in  fields 
of  nanotoxicology  and  ENM  risk  analysis  would  be  ideal. 

In  light  of  these  findings,  it  is  recommended  that  future 
revisions  of  TEARR  and  other  relative  risk  ranking  tools 
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involving  ENMs  be  able  to  tap  into  publically  accessible 
databases,  such  as  the  Nanomaterial  Registry,  in  order  to  be 
able  to  update  the  underlying  data  sets  on  EHS  implications 
of  ENMs,  potentially  in  real  time.  As  significant  research 
efforts  are  currently  focused  on  the  collection  and  curation 
of  data  on  ENMs  and  their  behavior,  risk  ranking  tools  such 
as  TEARR  would  benefit  from  accessing  these  publically 
accessible  databases.  Separate  analyses  that  assess  “new” 
levels  of  uncertainty  if  additional  data  were  required,  such 
as  Value  of  Information  methodologies  (e.g.,  Linkov  et  al. 
2011),  may  also  be  beneficial  when  considering  such 
updates  based  on  current  knowledge.  In  addition  to  these 
recommendations,  it  is  also  recommended  that  increased 
funding  is  made  available  to  support  the  development  of 
decision  support  tools  for  emerging  technologies,  including 
those  focused  on  risk  ranking,  which  specialize  in  condi¬ 
tions  of  extreme  uncertainty.  Given  the  ever-increasing 
pace  of  ENM  development  and  incorporation  into  various 
products  applications  as  well  as  the  extreme  challenges  to 
develop  the  data  required  to  satisfy  quantitative  risk 
assessment  approaches,  decision  support  frameworks 
which  focus  on  handling  extreme  conditions  of  uncertainty 
in  a  transparent,  structured,  and  robust  manner  are  critically 
needed,  particularly  for  decisions  regarding  emerging 
technologies  in  the  twenty-first  century. 
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